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Introduction 


Concept maps and other similar graphical representations, 
where concepts are linked to each other, are widely used in the 
teaching and learning of physics (Ingec¢, 2008; van Zele, Lenaerts 
&Wieme, 2004; Yin, Vanides, Ruiz-Primo, Ayala & Shavelson, 2005), 
chemistry (Nicoll, Francisco & Nakhleh, 2001) and biology (Kinchin, 
De-Leij & Hay, 2005; Kinchin, Hay & Adams, 2000), as well as in 
different levels of education from the elementary to advanced 
levels. Concept maps can also been used as tools of assessment 
and evaluation and as tools of monitoring development during 
learning (Ruiz-Primo & Shavelson, 1996; van Zele et al., 2004; Yin 
et al., 2005). 

The structure of concept maps indicating of the quality of 
students’ knowledge has been the focus of many studies. Attention 
is then paid to how concepts are interconnected locally or to what 
the “semantic fields” are that are provided by these connections. 
Consequently, counting of the links can be used to quantitative 
classification and the scoring of concept maps (McClure, Sonak & 
Suen, 1999; Nicoll et al., 2001; Novak & Gowin, 1984; Ruiz-Primo & 
Shavelson, 1996). Recently it has been suggested that both good 
understanding and the high quality of students’ knowledge is 
reflected as structures, which are tightly connected sets of several 
nodes and web-like (Derbentseva, Safayeni & Canas, 2007; Ingeg, 
2008; Liu, 2004; Kinchin et al., 2000, Kinchin et al., 2005; Safayeni, 
Derbentseva & Canas, 2005; Vanides, Yin, Tomita & Ruiz-Primo, 2005; 
van Zele et al., 2004; Yin et al., 2005). Such qualitative methods 
for analysing the concept maps have thus revealed that global 
topological features which are chain-, sooke- or web-like carry 
important information about the quality of knowledge represented 
in the maps (Kinchin et al., 2000; Kinchin et al., 2005; van Zele et 
al., 2004). 

In this study, a new quantitative method is introduced to re- 
veal the important structural feature of connectedness in concept 





ro CS 
RS 


ISSN 1648-3898 


Abstract. /n this study, a parsimonious 
set of quantities is developed to charac- 
terise and evaluate the quality of concept 
maps. It is shown here how such a set 
can be based on measurements of the 
connectivity, clustering and cohesion of 
concept maps. A structural model of the 
data is introduced so that concept maps 
can be evaluated and classified by using 
only two quantities called here richness 
of content (D) and quality of structure (q). 
The model is used in monitoring changes 
in the concept maps (N=33) consisting of 
two sets of maps (initial and final ones), 
made before and after teaching. Such an 
approach gives access to monitor changes 
of the concept maps during a teaching 
sequence. 
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maps and on this basis, monitor changes in the pre-service physics teachers’ concept maps when maps 
before teaching and after teaching are compared. The key structural features identified here are related 
to the local interconnectedness in form of the clustering and to the global interconnectedness in form 
of the cohesion. Clustering (denoted by C) means a property, where concepts form small and local in- 
terconnected cluster. The cohesion (denoted by ) is closely related to the overall interconnectedness 
in large scale and navigation due to longer cycles. 

The objective is to develop suitable quantities for the classification of the concept maps, so that 
the classification is based on the richness of content and the quality of structural features. This approach 
is based on the normalised structural variables C and QO introduced here. A structural model of the 
data, which allows us to predict the expected deterministic increase of the clustering and cohesion, is 
introduced. The analysis of the variations in the sample allows then detection of individual cases which 
are better than expected. These quantities can be used in classifying and monitoring the changes in 
student's concept maps, which represent the students’ ideas of the relations between physics concepts. 
This eventually enables the description of the structural quality (denoted by q) of the concept maps in 
greatly condensed form by using only one variable termed the structural quality and makes possible 
to monitor the students’ development on organising the knowledge by using the maps (and thus sup- 
posedly also conceptual development) during the teaching. 


Methodology of Research 


The concept maps discussed here were constructed by following design principles where links 
represent either modelling procedures or procedures of quantitative experiments. The most essential 
difference between the sample of maps discussed here and the traditional concept maps is that, in 
our case, the links are not propositions but based on a more detailed description, for example, how a 
certain experiment or model is used to motivate the introduction of a new concept. A justification was 
required for each link, so that students were asked to write down a short description. In what follows, 
it is important to remember that in order to establish a link students must be able to justify the link so 
that it is ontologically correct (correct entities are used), epistemologically acceptable (acceptable at- 
tempt is made to justify the reasoning and logic), and methodologically plausible (experiment can be 
made, model involves correct mathematics). This means that every link represents knowledge, which in 
principle is usable for purposes of teaching, i.e. “usable knowledge”. Therefore, all links can be taken as 
“correct” ones. Other aspects of the design principles are discussed in more detail elsewhere (Koponen 
& Pehkonen, 2010; Nousiainen & Koponen, 2010). 

The concept maps constructed on basis of the above mentioned design principles have been 
applied in physics teacher education in a teacher preparation courses for third year pre-service phys- 
ics teachers during two academic years 2009 and 2010. Students were asked as a course task to draw 
concept maps from given subject areas of physics. The goal of the course is to form a holistic overview of 
physics by producing concept maps and writing down descriptions about the linking procedures. Within 
the course “new physics” is not really taught but the focus is on the organising of concepts and binding 
together the pieces of knowledge students already had. Students worked on their own representation 
but collaboration between students were encouraged during process. The feedback from the students 
on use of the concept maps has been appreciative, many students noting the advantages of being able 
to visualise complex conceptual connections by using the concept maps. 


Sample of Research 


The sample of concept maps (N=33) to be analysed here were produced by third-year university 
students during physics teacher education courses in academic years 2009 and 2010. The courses were 
similar, of 7 weeks duration both. All the maps were drawn from the point of view as to how teacher 
students conceived the concept and laws of electromagnetism as they were introduced in the first-year 
university course, and how they thought the concepts and laws are connected through the aforemen- 
tioned procedures. The number of the concepts were limited to n=34 most central concepts, while 
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students were free to introduce as many links as they found necessary. An example of the concept map 
with n=34 nodes is shown in Figure 1. This concept map is an example of the most extensive and best 
ones. In these kinds of concept maps the most well connected concepts are typically linked to 3-5 other 
concepts. Some connecting procedures are given in Table 1 with numbering in reference to Figure 1. 
It should be noted that the numbering of links denotes the sequence of steps in the construction of 
the concept map. 
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Figure 1: Anexampleofaconcept map: concepts are illustrated as boxes, laws and principles boxes 
with thick borders and conceptual models of a real system are shown as ellipses. Linking 
procedures are either models (M) or experiments (E). 
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The original concept maps drawn by students each have a personal graphical layout (i.e. a way 
to draw the map), which often makes it difficult to see important topological features or which may 
misguide the attention away from the relational structure to mere appearance. The visual appearance 
of maps was made comparable by using systematic re-drawings of the maps so that the same rules 
for ordering the nodes are used in all cases. In graph theory this is called embedding of the graph (i.e. 
here the map). Here graph-embedding method called the spring-embedding has been used, which is 
obtained when each link is assumed to behave like a spring and which minimises the total energy of the 
spring system (for details see Pemmaraju & Skiena, 2006). Examples of the maps in the spring-embedded 
form are shown in Figures 2 and 3 for maps with n=34 nodes, respectively. 
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Table 1. Some examples of modelling and experimental procedures from the concept map illus- 
trated in Figure 1. 


Modelling M11: Definition of electrostatic voltage as a potential difference. 

Procedures M M21: Magnetic potential energy identified when turning an electrical circuit in magnetic field. 

(only part) Interpretation through analogy to a particle’s potential energy in electric field. 

Experimental E5: Coulomb’s experiment and quantification electrostatic force. 

procedures E E13: Capacitance and capacitor law defined using electrostatic generator 

(only part) E23: Ampere’s experiment and quantification of current and force between current carrying 
wires. 





Figure 2: Three examples of spring embedded concept maps with different level of clustering show 
that somewhat similar structure can be found in each case. 


The structural features of the maps -— clustering-like interconnectedness locally, cyclical intercon- 
nectedness in global scale and the overall web-like character — displayed in Figure 2 are those essentially 
brought forward by qualitative analysis of concept maps (c.f. Kinchin et al., 2000; Kinchin et al., 2005; van 
Zele et al., 2004). All these qualitative notions based on visual inspection will now be operationalised 
and made quantitative. 


The Method of Analysis 


The structure of the concept maps can be analysed as network structures the analysis of which 
takes place in three steps: 

1. The qualitative features of interest are transformed in a form amenable to quantitative 
analysis. This part is carried out by using the graph-theoretical approach. 

2. A quantitative analysis of measured properties is carried out. The purpose of quantitative 
analysis is to reveal those measurable properties and their systematic dependencies which 
allow us to classify the maps on the basis of the richness of their content and the quality of 
their structure. 

3. On the basis of the classification a quantitative measure for the structural quality is intro- 
duced. 


The qualitative structural features of primary interest are clustering (local connectedness) and 


cohesion (global connectedness). Clustering means here a property by which concepts tend to form an 
interconnected local cluster. Cohesion means a connections provided by larger clusters, which provide 
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paths to navigate in structure and which hold the structure together. These features can be operationa- 
lised by defining the following variables for each node k locally (Kolaczyk, 2009): 

1. The degree D, is the number of links connected to a given node k. The degree D, contains the 
incoming and outgoing links. Therefore, it directly measures the connectivity of the node. 

2. Theclustering coefficient C, measures the local interconnectedness of concepts through as 
the ratio of triangles to the all triply connected neighbours around a given concept. C, obtains 
values between 0 and 1, where 1 corresponds to the maximal number of triangles obtained 
by connecting all neighbours. The clustering is a kind of “family tie” measure. 

3. Thecohesion ( is defined so that it will be related to importance of longer cycles. The main 
interest is in longer cycles consisting of combinations based on triangles, because they pro- 
vide cross-links between nodes, i.e. short-cuts. These are essentially odd cycles, i.e. 3-cycles, 
5-cycles etc. Therefore, cohesion is defined as the ratio of the number of odd-cycles to all 
cycles. The cohesion then directly relates to the overall connectedness of the network and 
to interconnectedness in large scale and navigation in providing cohesion. 


The values of the variables D, C and Q on a given concept map (network) can be calculated 
when the variables a, (adjacency matrix a) are known. Their detailed mathematical definitions are given 
in Appendix A. 

The local analysis of the concept maps produces a great amount of detailed data, which need 
to be reduced for the practical purposes of classification and comparison of different concept maps. 
Of the average degree of nodes, called degree D, is then the basic variable related to the connectivity 
and the number of the links (richness of content), C to clustering (local interconnectedness) and Q to 
cohesion (local interconnectedness). Because the properties of being hierarchical and clustered are 
important qualitative features, it is clear that in the structurally good concept maps all the quantities C 
and Q need to have large values. 

In order to analyse the systematic dependencies and the variations around the mean the variables 
C and Q on Dare regressed, but it should be noted that the dependence (regression) is not linear. The 
regression means that the connectivity or the richness of content D is selected as an independent (or 
explanatory) variable, while the set {C; 0} consists of the dependent (or response) variables. This selection 
is motivated on the basis of a theoretical understanding of the expected dependencies and in practice the 
selection is the first step in the structural modelling of the data (see e.g. Mulaik, 2009). Such a structural 
modelling aims to condense the variables in a form of multivariate distribution (probability density) P(C; 
Q) which tells the frequency (probability) to obtaining a given set of values {C; QO}. This probability density 
function can then be used as the basis for defining the structural validity so that the values larger/smaller 
than the average indicate structurally better/poorer than average maps. In practice, direct measure for 
that is given by the cumulative distribution function (CDF) ®(C,Q) = Prob (P;C'< C3O'<Q ) of 
probability density function P in defining the structural quality in the form 


g = D(C, Q) — B(0, 0) 


The quality g then plays a direct role in interpreting how much the map with the given values of C 
and O deviate from the expected average map. Values g < 0 indicate structurally poorer than average 
maps while g > 0 indicates structurally better maps. Of course “structurally good” is then simply a state- 
ment that clustering and cohesion are all above average values, while “structurally poor” means that all 
values of structural variables are below average values. Similarly, “structural quality” refers only to these 
properties. The structural quality is completely independent of D and therefore g and d represent truly 
independent dimensions; structural qualities and richness of content. 


Results of Research 


The sample of the concept maps studied here consists of two sets of maps, but in both cases the 
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design principles are the same. The first set is the maps made by students before group discussions, 
where maps are compared and evaluated. This set is called the initial maps. The second set is done after 
the group discussions, with the purpose of improving the maps. This set is called the final maps. Some 
examples of the initial and final maps in the spring-embedded form are shown in Figure 3. 


A initial r B initial 





A final B final 





Figure 3: Concept maps n=34 drawn in the spring-embedded form, initial maps have D=2.23 Q=- 
0.13 (map A) and D=2.88 Q=-0.31 (map B) and final maps D=2.65 Q=-0.06 (map A) and 
D=3.29 Q=0.33 (map B). 


The measured values of the variables C and ( as regressed on the values of D are shown in Figure 
4 for the sample consisting of N=33 concept maps. Analysis of the distribution shows that the distribu- 
tion of C and (Q can be represented by multi-normal distribution with standard deviations o,, =0.038 
and o,, =0.048 respectively (see Appendix B). The correlation coefficients between variables are given 
by n=0.82. The goodness of fit is tested for the marginal normal distributions by using the standard 
y’-test (Ruskeepad, 2004), yields p-values 0.94 indicating very high goodness of fit. 
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Figure 4: Theclustering Cand cohesion 2 as they depend on scaled degree d. Fits are given by Eq. 
(1) with parameters k =3.77 and a.=0.75 for the clustering C, and k,=1.29 and a,=1.40 
for the cohesion 0. 


Clustering. The results in Figure 4 for C show that also clustering increases with increasing D in 
initial and final maps alike. This reflects the fact that, if concepts A and B and B and C are related, it is 
very probable that also concept B and C are related (the transitivity condition), therefore giving rise to 
triangular patterns. The high clustering can be thus taken as a sign of tight interconnectedness in close 
neighbourhood of given concept. 

Cohesion. From the results in Figure 4 can be seen that the cohesion (Q of the concept maps increases 
rather rapidly with the increasing scaled number d of connections per node. Cohesion measures the 
networks’ stability to stay in one piece; that means, the larger the cohesion is, the larger is the prob- 
ability that the network cannot be divided into two subsets of nodes where nodes in the given subset 
are not connected. 

The structural quality g of the concept map, based on CDF of the multivariate distribution P and 
defined in Eq. (4), is shown in Figure 5. The quality of initial concept maps is shown by stars while the 
quality of the final maps is given by bullets. It should be noted that average structural quality corresponds 
case q=0. The concept maps with the poorer than average quality have negative values of g while those 
which have better than average quality have positive values. From the result it is seen that g indeed 
classifies the different maps so that there is a continuum of values but g does not depend on d. 
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Figure 5: The structural quality q of the concept maps for maps with scaled degree d. The initial 
concept maps are marked by stars the final maps by bullets. 


The connectivity d and the quality g represent very different qualitative properties of the maps. 
Taking into account that all links are procedural connections, based either on experimental or modelling 
procedures, d simply represents the richness of the procedural knowledge that students have at their 
command. Therefore, d and g represent independent dimensions of properties; richness of content 
and quality of structure. Having now the two quantities d and q which can be used to characterise the 
structural quality of the concept maps, they can be used to monitor the change by using these quanti- 
ties. The differences can be formed Aq=q,_-q,_..,andAd=(d,__-d._..)/d,._, where Adis scaled to give the 


relative change. These changes from the initial to final maps are shown in Figure 6. 
0.8 
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Figure6: Thechange in quality in the concept maps with relative change of scaled degree d. 


The figure 6 describes the changes in the structure of concept maps obtained during the learning 
(i.e. learning outcomes) and show that there is notable development of structure, and thus expectedly 
also in students’ conceptual understanding. The relative change in the richness of content (in x-axis) 
means that students have managed to integrate new content in their concept maps. For example 
relative change of 0.5 means 50% of new information as final map is compared to the initial one. The 
most important result is that most of the students are located right upper quadrant in Figure 6 which 
tells us these students really have made progress. Of these students some have achieved substantial 
changes in quality of the structure but only small relative changes in the richness of content. There is 
a small group of students, which have reached substantial increase in richness of content but small 
changes in quality of structure. The results seem to indicate that it is not easy to perform well in both 
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aspect (increasing the content and improving the structure) and that the most productive learning ap- 
parently takes place in the region where large changes of structure take place but changes in richness 
of content are moderate. 

If there are large changes in richness of content, change in structure remains moderate or the 
structural quality can even become poorer as in the lower right quadrant. It might be that students have 
drawn initial map in too optimistic way (i.e. drawn too much links, maybe based on loose association) 
and realised afterwards that they are not able to justify all links and thus have deleted some, with the 
result of deteriorating the structure. In some cases, located in upper left quadrant students have appar- 
ently improved the structure simply by“pruning’, by reducing the richness of content in favour of better 
connected structures. Finally, the lower left quadrant represents cases where all has gone amiss - the 
content is impoverished and structure deteriorated. 


Discussion 


The most important finding of this study is that structural variables describing the topology of the 
concept maps can be used as basis to define unambiguously the quality of the structure. The structural 
validity of the analysis concerns the question of whether or not the variables provide information on the 
structural aspect. In regard to the clustering and cohesion, this question is answered nearly trivially by 
noting that: 1) the variables are constructed so that they operationalise the qualitative structural features 
of interest, 2) the definition of variables is mathematically exact (see Table A1 in Appendix A), and 3) the 
analysis itself is based only on these well-defined topological measurements of the concept maps. 

The content validity is to a large degree resolved by the notion that students need to justify each 
link and explain its (procedural) content in written report coming with the map. Although the depth 
and thoroughness of these explanations vary, they are always acceptable (i.e. correct models and 
experiments) from the point of view of physics content knowledge. Therefore, the number of links is 
directly related to the richness of correct content (connectivity), while the good structure is related to 
the ability to organise this content. 

The remaining question is the reliability of the analysis, concerning the statistical significance of 
the averaged values of variables (i.e. local variables averaged over the whole map). It has been shown 
in this study that the suitably scaled data can be described quite reliably by means of a multivariate 
normal distribution, which suggests that the residuals form a set of data, which is distributed inde- 
pendently, identically and normally, that is, it represents the so-called heteroscedastic data set. The 
possibility to have a data model, where the structural model describes the expected mean and the 
probability density P the deviations from the mean, forms the basis on which to attribute one quantity 
q to each concept map so that it describes the quality of the map in terms of the probability measure 
of the expected deviation. 

The research carried out here has concentrated on finding structural model to describe concept 
maps and to develop unambiguous measures to monitor the structural change. However, from point 
of view of learning and teaching the main focus is the development of method to monitor conceptual 
change and development in learning during a teaching sequence. The mutual changes in relatedness 
of concepts is directly reflected in the structure of concept maps, which suggests that such changes 
can be used in monitoring conceptual development, as has been discussed for example by di Sessa and 
Sherin (1998) and Liu (2004). The development in richness of content and quality of structure are both 
important aspects when evaluating students’ concept maps. Therefore, the richness of content and the 
quality of structure as introduced here open up the possibility to monitor changes in conceptual struc- 
tures and make inferences of the students’ development during a teaching sequence. The results show 
that in the most cases there is positive development where both richness of the content and quality of 
structure are improved, but only in one or another dimension. Progression in both dimensions (quality 
of structure g or richness of content d) is not easily reached and it might well be too complicated task 
for the students. The moderate changes either in quality of structure g or richness of content d probably 
means that teaching and learning happens near students’ the zone of proximal development (ZPD) as 
described by Vygostky (1978). According to Vygotsky's theory, teaching and learning is productive and 
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meaningful if its goals are set so that they are a bit more demanding than the goals easily achieved by 
the students in their starting position. In that case teaching promotes development of actions, which 
are soon developing, i.e. actions which are in the ZPD. Thus, in co-operation and supervision (guided 
learning) students are able to solve more complex tasks than alone. Apparently in this case, improving 
either structure or content is within the ZPD whereas, improving them both overshoots ZPD. 


Conclusions 


The structural variables describing the topology of the concept maps can be used as basis to 
define the quality of the structure. A quantitative method of analysis of concept maps is introduced 
here, which provides tools for analysing relevant complex features with quantitative accuracy and for 
assessing the quality of the concept maps on this by using valid and reliable measurements. It is shown 
that such an approach gives access to monitoring change of the concept maps during the teaching, and 
that the changes brought forward can be meaningfully seen as conceptual development or change. 
These notions encourage thinking that the methods developed here provide fruitful starting point for 
monitoring learning outcomes and for their assessment and evaluation. 
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Appendix A 


In order to make the detailed analysis of the concept maps amenable first the variables are defined. 
In a map (network) of n nodes, the variables qj form a nxn dimensional so called adjacency matrix a. 
The independent variables are binary variables a,, which indicate the connections between nodes / 
and j so that if nodes are connected, then q=1 and if there is no connection, then a,=0.The dependent 
variables are the number of links of each node (connectivity) and the clustering and cohesion as defined 
in Table Al 


Table A1. Mathematical definitions of variables for measuring the topology of the networks. Note 
that in the definition of O the numerator is the number of odd cycles and denominator 
the number of all cycles S, going through node k. The subscripts k refer to the kth node. 
The adjacency matrix is given by a and it has elements a. 


Variable 


Definition 
Connectivity Dk > (a, + a;) 
Clustering Ck 2. jE Uj | en jp Cie gy 
(>,(@"),, (2i+1)!) 
Cohesion 0 A 


(2) yi!) 


The classification of the maps, definition of the quality of the maps and ultimately the monitoring 
of the changes in the quality will be done, it what follows, by using the values of C and 0 averaged over 
the all nodes. 
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Appendix B 


In order to make the detailed analysis of the concept maps amenable first the variables are defined. 
The systematic dependence of the averages <X> of variables X={C; QO} on D in the studied sample of 
maps is modelled by the structural equations of form 


(x)=(4k,d?*), d=D/D,-1 


In the Eq. (1) parameters kK and a, define the nonlinear dependence on suitably scaled param- 
eter d=D/D,-1, with D, as a scale parameters. Note that the model is valid only for D > D,, where value 
D, = 2.0 is related to the threshold when maps begin to have an appreciable number of 3-cycles. All 
parameters are defined empirically, by using least squares fit to the pool of data. The justification of 
the structural model is given a posteriori by inspection of the residuals of the data. In the best case the 
structural model helps to represent the data so that the residuals x = X - <X> are normally, identically and 
independently (NIID) distributed. In this case the so-called heteroscedasticity condition of the residu- 
als is fulfilled, which is needed in order to carry out unbiased and reliable statistical analysis (Johnson 
& Wichern, 2007). When the NIID condition is fulfilled the normal distribution can be used to describe 
the data, with average values given by the structural model and with variances obtained from the het- 
eroscedastic distribution of residuals. In this case the distribution of variables X={C; O} is then given by 
the multivariate normal distribution. 


P[X;=] = Z-‘expl- X75" X] 


where X is a vector of variables and Z is the normalisation factor. The covariance matrix = is given by 


2 
5. AS Sq 


7 2 
NS ge So 


— —<— 


where standard deviations of C and Q are oC, and o,, respectively. The correlation coefficients between 
variables are given by n. When the values of the variables are distributed according to Eq. (2)-(3) it is 
possible to tell directly what the probability P(C;,Q) is for obtaining the set of given values {C;Q}. 
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